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ABSTRACT 



The Assessing Quality on Digital Reference team conducted 
two site visits at Florida Libraries in August 2001. The objectives of these 
visits were to: document how digital reference services are currently being 
planned for, delivered, and evaluated in libraries; understand how "quality" 
in digital reference is defined, in these environments; identify current 
issues in the provision of digital reference surrounding technology, 
management, staffing and training, and costing and economic concerns; and 
determine the evaluation needs of libraries regarding digital reference 
services. Data were collected in in-depth interviews with key informants at 
each library. Each library chose the personnel they felt could best inform 
the research, typically including both managerial and professional 
perspectives. Each library received a copy of the questions to be covered in 
the interview in advance of the site visit and each was well prepared to meet 
with the interviewers. In addition to the pre-formatted questions, interviews 
included opportunities for respondents to raise issues and add topics to the 
agenda. In the course of these interviews several trends were documented that 
are developing as a common thread in the academic literature and in data 
gathered on site visits at libraries across the United States. Provided here 
are the site visit reports concerning the provision of digital reference 
services at the State Library of Florida, Bureau of Library and Network 
Services and the Tampa -Hillsborough County Public Library System. These 
reports are followed by a description of how these findings fit into a larger 
effort to assess the quality of digital reference, produce evaluation tools, 
and determine how best to integrate evaluation efforts into digital reference 
services. (AEF) 
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Assessing Quality in Digital Reference Services Site Visit Reports 



The Assessing Quality on Digital Reference team conducted two site visits at Florida 
Libraries in August 2001 . The objectives of these visits were to: 

• Document how digital reference services are currently being planned for, 
delivered, and evaluated in libraries. 

• Understand how “quality” in digital reference is defined in these environments. 

• Identify current issues in the provision of digital reference surrounding 
technology, management, staffing and training, and costing and economic 
concerns. 

• Determine the evaluation needs of libraries vis-a-vis digital reference services. 

Data were collected in in-depth interviews with key informants at each library. Each 

library chose the personnel they felt could best inform the research, typically including both 
managerial and professional perspectives. Each library received a copy of the questions to be 
covered in the interview in advance of the site visit and was well prepared to meet with the 
interviewers. In addition to the pre-formatted questions, interviews included opportunities for 
respondents to raise issues and add topics to the agenda. 

In the course of these interviews several trends and issues were documented that are 
developing as a common thread in the academic literature and in data gathered on site visits at 
libraries across the United States. These are: 

• Many digital reference services have grown organically out of the library’s 
perceived need to provide digital reference service and not as the result of detailed 
planning. 
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• The volume of digital reference questions is generally low, but can be increased 
with outreach efforts and the increased visibility offered by advertising and 
formal marketing efforts. 

• The low volume of questions has kept the need to address many management, 
staffing, and training issues at bay. If these services are expanded, these issues 
will come into focus and formal policy, procedure, and evaluation strategies will 
be needed to support them. 

• Libraries continue to be concerned about the need to provide reference services 24 
hour a day, seven days a week. It is valid to question whether this service is 
necessary for every library type and whether the library’s designated user group 
or service area will in fact benefit from it. 

• Many predict future movement toward increased accountability in providing 
costing data on a program basis. To do this libraries will need to develop new 
ways of apportioning costs and benefits if digital reference services are to be 
accurately described. 

• Efforts at evaluating digital reference have been limited and rudimentary. 
Emphasis has been on volume statistics and the need to measure user satisfaction. 
Evaluation plans need to be formalized and new metrics and measures developed 
to allow for the appropriate evaluation of digital reference services in libraries. 

Provided here are the site visit reports concerning the provision of digital reference 
services at the State Library of Florida, Bureau of Library and Network Services and the Tampa- 
Hillsborough County Public Library System. These reports are followed by a description of how 
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these findings fit into a larger effort to assess the quality of digital reference, produce evaluation 
tools, and determine how best to integrate evaluation efforts into digital reference services. 
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State Library of Florida, Bureau of Library and Network Services 
Digital Reference Site Visit 
August 14, 2001 

Introduction 

Project team members Melissa Gross and Ruth Hodges conducted a site visit on August 
14, 2001 at the State Library of Florida, Bureau of Library and Network Services. In-depth 
interviews were held with the Bureau Chief and the Head of Information Services. 

Access to the State Library of Florida’s email reference service, Ask A Librarian, is 
available on almost every page of the library’s web site. The address for the main page is: 
http://dlis.dos.state.fl.us/stlib 

A variety of internal documents relating to the email reference service were provided to 
the interviewers. These include copies of the: 

• Reference Survey Evaluation form used by the Bureau to evaluate customer satisfaction. 

• Florida Statute 25 7.261 concerns the confidentiality of customer records. 

• Public Service Measures 2000-2001 . 

• Copies of the main page and sample questions from the Reference Desk Transaction 
Database a tool developed and used by staff to save model and difficult questions. 

• Sample records. 

• A memo describing the performance standards for digital reference. 

• Samples of publicity materials. 

• The completed Question Guideline form that was sent to liaisons prior to site visit was 
collected. 

• Current draft, Email Reference Procedures Manual 
All documents related to this site visit are on file at FSU. 
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Selected Notes and Issues 

The State Library has provided email reference services since 1996. Ask A Librarian 
initially started as a link at the bottom of the State Library’s Government Information Locator 
Service (FGILS) web page. Since that time, the volume of electronic reference questions 
received has grown from 3,292 logged in 1997-1998 to 8,931 received in 2000-2001. The 
library currently receives and processes 400+ electronic reference questions per month. 

The volume of questions tends to increase during legislative session and to decrease 
during the summer when school is out. For about a half a year the searchable data base 
component of the Florida Government Information Locator Service was linked to 
MyFlorida.Com and served as their search engine. During that period the Ask A Librarian 
service had a high level of visibility and the volume of questions received increased to over three 
times the normal load. Many of the questions received were specific to the MyFlorida.Com web 
site. When this relationship ended, the volume of questions returned to the normal 400-500 
questions-a-month range. 

The library accepts all questions. When necessary, they will provide the requester with 
contact information to other agencies, institutions, etc. The library provides both fax (not as 
prevalent as email) and email information. Long distance users with lengthy calls are referred to 
the Ask A Librarian service. The library views email and face-to-face reference as being 
essentially the same service. They feel the professional skills the librarian needs are the same for 
both and their experience is that the time and effort to respond to email questions is similar to the 
time and effort required in traditional reference. 

Currently, the library has no formal, separate objective for providing digital reference. It 
is fully integrated into their concept of what reference entails in this environment. The library 
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developed Ask A Librarian as a response to developing technology. As a result, no needs 
assessments were conducted prior to offering it. Currently, there are no written policies or 
procedures for services but the library is in the process of developing policies based on past 
examples used. 

Digital Reference Process 

Two different web forms are provided on the web site for Ask A Librarian services: one 
for use by the general public and one specifically for employees of the State of Florida. 
Questions can also be submitted through a direct email address. The library considers the state 
of Florida employees as its primary clientele and makes special efforts to serve this group. 

Incoming questions are received at a general email address and then divided evenly 
among librarians and assigned at random, irrespective of whether questions are easy or hard. 
Some questions may be given to librarians with particular expertise or to librarians specifically 
requesting a question(s), but this is the exception. Librarians use print based resources, special 
collection materials, and electronic resources to respond to email reference questions. 

The turnaround time for questions per policy is three days. The actual time is usually 
less; some questions can be answered very rapidly using the Transaction Database. The 
Transaction database is a product developed and maintained by the librarians. It is used to 
archive previously asked questions that they wish to maintain access to. These are model 
question answer sets, questions that were difficult to answer, and frequently asked questions. 
This database can be searched to find answers to incoming questions, and is used to update the 
FAQ page. There are currently between 750 and 1,000 questions in the database. The librarians 
also have access to the sent file in Microsoft Outlook (the email software used for the service) if 
they wish to search there for a previously answered question. 
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The Ask A Librarian Service is offered in English only. It does not include any special 
user training or orientation. 

Marketing 

As stated above, Ask A Librarian is very visible on the library’s web pages. It is also 
actively promoted and publicized using both traditional library promotion strategies (such as the 
development of brochures), and in ongoing outreach efforts. One of their promotion strategies 
was the development of a CD-ROM give-away that won the LAMA PR All Star Award in June 
2001. 

There are two state agency liaisons charged with outreach as part of their regular duties. 
These employees make visits to the various government agencies to raise their awareness of the 
library’s services (including email reference) and attend the annual orientation of the legislature. 
Li addition, they staff a library booth at the Florida Library Conference and the local Technology 
Conference, held in Tallahassee annually in September at the Civic Center. They also conduct 
outreach from a booth in the capital during Library Day. 

Digital Reference Desk Users 

Although Ask A Librarian is available to the general public, the library views its primary 
obligations first to the state of Florida employees, second to state (public) libraries, and third to 
the general public. In providing these services, the library does not limit services to users by 
password, account, library card, or membership. Although the library does not reject non- 
affiliated users, it is primarily interested in answering questions that pertain to its special 
collections and the State of Florida. To this end, they do encourage the public to use their local 
library first, and a link to a list of Florida public libraries is provided on the State Library of 
Florida web site. 
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As indicated by the two web forms used for submitting questions, the library is especially 
concerned about tracking its primary user group, state employees. Their questions are analyzed 
to determine which agencies use the services and what parts of the collection they may want to 
know more about. This information is used to design future outreach programs. The State of 
Florida employee ’s web form collects the name of the agency; user name, various contact and 
mailing information; subject of search; place for request(s); and date needed. The general 
public's web form collects contact information, subject of search, and the question itself. Both 
web forms stress the importance of providing a telephone number in addition to an email address 
for the library to respond to. This is because the email addresses submitted are not always 
functional and because, if the nature of the question is complex or if a reference interview is 
needed, the librarian may elect to call the user to save time. 

To deal with privacy issues, library staff never forwards requests to other agencies or 
departments in state government, but instead provides users with specific contact information, so 
that they can do so directly if they wish to continue to pursue the information. This is done to 
comply with the Florida Statute 257.261 FS that guards privacy and confidentiality of user 
registration and circulation records. 

Evaluation 

A quarterly survey (Reference Survey form) is used to measure user satisfaction with 
reference services in general. This survey is interested in whether the information users receive 
is at or above their expectations, whether they receive help in a timely manner, and asks users to 
rate staff service behaviors on several dimensions (available, helpful, effective, courteous, 
understands the question). If the response to their question was a referral, the survey asks 
whether an appropriate referral was made, and lastly, the survey asks users to rate their overall 
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experience and the quality of the service. Users are asked to rate these dimensions as Poor, Fair, 
Good, Excellent, and No Opinion. The library does not collect information on user return rate or 
the number of new users. 

An email link is provided on all of the library’s pages soliciting user comments about the 
library’s web page and services. To date, the emails received via this link have been most 
helpful to the library in terms of helping them identify what they call “information literacy 
opportunities,” where users make them aware of other resources, but tend not to provide 
feedback about current services. Users mainly use the comments to report broken links, make 
solicitations for business opportunities with the library, and to voice complaints about 
government. Reference questions are occasionally submitted through this link and these are 
forwarded to the reference for handling. 

The main statistic the library collects is the total number of questions answered. They 
say that 1 00% of questions are answered. They do not do a formal assessment of the percent 
answered correctly, but the supervisor reviews the question/answer sets for accuracy and 
completeness as part of employee assessment. 

The interviewees expressed interest in categorizing questions by type (ready reference, 
research, etc.). The number of hits to the web site is currently being captured, but the library is 
also interested in knowing how many people view its web site and find what they want without 
having to engage the Ask A Librarian service for assistance. They are not particularly interested 
in demographic data. 

During the next legislative session, the library is planning to conduct a focus group to get 
feedback on new wireless access to the library’s web page that is currently under development. 
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They want to target their frequent users to find volunteers to test and refine this new mode of 
service delivery. 

Digital Reference Service Staffing and Training 

The reference desk is staffed nine hours a day from 8:00 a.m. to 5:00 p.m. Monday 
through Friday and Saturday from 9:00 a.m. to 3:00 p.m. (email questions are answered on 
Saturday only as time permits). All reference librarians answer both traditional and email 
reference questions. There is no separate digital reference staff, but only librarians with at least 
one year of reference experience are asked to perform reference via email. 

Newly hired librarians are trained to perform email reference as part of their general 
training and orientation process. Orientation lasts for several months. During this time the new 
librarian sits with an experienced librarian at the reference desk and their email question and 
answer sets are reviewed and commented on until the librarian is ready to go solo. 

Reference performance is measured and evaluated as part of the duties of the reference 
librarian. That is, librarians have an annual performance appraisal where accuracy and 
completeness in answering questions are reviewed. The determination of what accuracy and 
completeness means is somewhat subjective, but librarians are provided with model answers and 
have access to the question/answer sets of their peers. Librarians are aware of the performance 
standards and expectations. All librarians are expected to answer email reference questions in 
three days or less and to answer all of the email reference questions they receive. 

Economic/Cost Issues 

There is no separate budget line established for email reference. Reference services at 
the State Library are supported with state funds and the cost of reference is not isolated or 
separately accounted for. Funds are apportioned under headings like “materials” or “personnel” 
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and not by service or program. Perhaps because of the way funding is handled, the library has 
not investigated the question of what a reference transaction costs. Thus, there are no statistics 
kept or measures employed to determine cost effectiveness and/or to perform a cost/benefit 
analysis. 

Some library goals are tied to its budget in order to meet Performance Based Budgeting 
requirements. These goals tend to be things like number of items circulated, number of reference 
questions answered, and reports of user satisfaction. This makes it clear why statistics on 
volume and the quarterly user satisfaction survey are so important. Meeting and exceeding these 
goals can be rewarded and currently the library is using funds gained this way to extend service 
hours. There have been no noticeable effects of providing email reference on other library 
expenditures. During the period when volume tripled, OPS money was available to hire two 
students for fall term to pick up the overload. 

Technology 

Hardware/software needed to support DR service include: Microsoft SQL version 7.0, 
ColdFusion Professional Server 5.0, TextPad Editor, Dell PowerEdge 2300 Server (housed at the 
State Library of Florida), and Microsoft Outlook and additional web pages hosted on servers at 
the Department of State Central Computing Facility. Generally, the library appears satisfied with 
the email technology they are using. Hardware/software needed to access DR service includes a 
browser, PC, and email access. 

Upgrades or changes in hardware/software are in the works only in the sense that the 
library’s technology needs are a part of the Division’s long-range technology plan. The library 
has investigated using chat software to provide digital reference, but they are not convinced that 
it will add any real value to what they do. Because of the nature of the users they serve, they 
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have let go of the idea that they need to provide 24/7 services, especially since they do not have 
24/7 access to their own building and facilities. 

Library staff are currently involved in developing access to the library’s website using 
wireless markup language (WML) and instant messaging. Their vision is that when the 
Legislature is in session, legislative aids and others should be able to access the library’s 
databases and reference services from the floor using their PDAs. Although customers have not 
asked for wireless service, the library views having the capability to do this as being proactive. 
They are planning to use focus groups during the next legislative session and perhaps survey 
them as they begin to use the product. 

As with the development of email reference in this enviromnent, this service is being 
generated out of the interests and creativity of current staff working without a separate budget in 
an exploratory plan that is dependent on the staffs own motivation and the eventual responses of 
the target users. 
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Tampa-Hillsborough County Public Library System 
Digital Reference Site Visit 
August 22, 2001 

Introduction 

Team member Melissa Gross conducted a site visit with library staff at the Tampa- 
Hillsborough County Public Library System on August 22, 2001. Interviews were performed 
with the Supervisor of the Electronic Reference and Information Department (ERI), the Principal 
Librarian from Training and Service Development, and the Principal Librarian from Materials 
Services. An additional telephone interview was performed with the director of the library on 
July 13, 2001. 

The afternoon of the site visit included a tour of the Electronic Reference and Information 
Department and informal observation and interaction there. 

Access to the QuickAnswers! Service is from the library’s home page at: 
htt p://www.thpl.org/ from the QuickAnswers! information page http://www.thpl.org/thpl/eri/, or 
it can be directly accessed at: http://www.thpl.org/thpl/webmaster/forms/askalibrarian.html for 
web form submissions or via email directly to answers@thpl.org. 

Also available online are two sites that provide information about QuickAnswers!, the 
QuickAnswers! Information Page at: http://www.thpl.org/thpl/eri/ and the Ask-A-Librarian Tips 
page (http://www.thpl.org/thpl/eri/askalibrarian tips.html) that outlines policy and procedures 
for the service. 

A variety of internal documents on training for digital reference, personnel evaluation, 
and internal procedures were provided to the interviewer. These are currently on file at the 
Florida State University and include: 

• QuickAnswers! data sheet. 
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• Canned comments and responses the librarian’s use in composing reference 
responses. 

• ERI Training Program Sheets. 

• Guidelines for answering email questions (includes issues of format, content, and 
style). 

• Instructions for answering questions submitted through the E-Library kiosks. 

The supervisor of ERI offered to photocopy and send one year’s worth of statistics describing the 
volume and question type of email reference queried received in the first year of the service. 
When this data is received it will be reviewed and filed with the other materials pertaining to this 
report. 

Selected Notes and Issues 

QuickAnswers! is a centralized telephone and email reference service serving the entire 
Tampa-Hillsborough County Public Library system. It is physically located at the John F. 
Gennany Public Library and maintains the same hours of service as the public library (Monday 
through Thursday 9 am- 9pm, Friday 9-6, Saturday 9-5, and Sunday 10-6). The emphasis of 
the site visit was on the provision of email reference service and not on the call center approach 
to telephone reference. This section overviews the background and operation of this service to 
date. This service appears to be entering a phase where it will be considering new delivery 
modes and methods of evaluation, but these are largely externally driven, either by mandates 
from the county or by participation in consortium agreements. 



3 

ERIC 



Page 15 



16 



Gross and McClure, August 2001 



Assessing Quality in Digital Reference Services Site Visit Reports 



QuickAnswers! Background 

The QuickAnswers! email reference service is about two years old. It was developed to 
funnel patron emails away from the web master, who was redirecting these email requests to the 
reference librarians. It is clear that the service was thought through and carefully planned, but 
the person in charge of this effort left the library before the service was launched and no 
documentation from the initial planning is currently available. Analysis of the web form 
interface indicates that the person who designed this service was thinking about evaluation 
issues. The web form asks detailed questions about the users including name and email address, 
where they live, their ages, which library they use, grade level (if student), what type of question 
they have, sources they have already consulted, whether they are repeat users, and the text of 
their question. 

Unfortunately, if an evaluation plan for the service was developed initially it has been 
lost and no formal evaluation of the service has been performed since its inception. Sadly also, 
the data bank of question and answer sets collected over the past two years are unavailable for 
analysis as the Electronic Reference and Information Department (ERI) routinely deletes them in 
an effort to “protect user privacy.” In what seems a somewhat contradictory move, the ERI is 
currently deleting the user’s questions, but preserving their email addresses in anticipation of 
performing a user satisfaction survey. Other libraries documented in the literature routinely 
preserve their question/answer sets and remove all identifying information in order to preserve 
confidentiality. 

The collection of statistics has been fairly rudimentary. Volume (number of questions) is 
tracked and for a short period of time some effort was put into analyzing question type in order 
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to inform collection development efforts. When system wide reporting standards changed, this 
analysis stopped. 

Email Based Reference Services 

Email reference questions may be submitted either via web form on the library’s web site 
or by using a direct email address. There are also several E-Library kiosks in areas not served by 
branch libraries and these kiosks provide reference service to users through the library’s web 
page. Questions received through the email address do not include the demographic or other 
information requested on the web form. The department supervisor estimates that about 10% of 
the email questions are received this way. Pine email is the standard for the county and there are 
no plans currently to upgrade this software. The library reports being fairly satisfied with Pine 
except that it is difficult to send attachments using this software and it hampers their ability to 
use attachments to send information back to users. This situation is somewhat ameliorated by 
the facility in several of the subscription databases the library uses to email articles directly to 
patrons from the databases. 

All email questions are received by ERI at one email address and then the system 
distributes them to all of the ERI librarians. ERI librarians are expected to read all incoming 
questions and to review all outgoing answers. They “claim” the questions they want to work on 
informally and verbally among themselves. In some cases, more than one librarian will work on 
a question, especially if it is difficult or research oriented. 

Evaluation 

ERI receives 100-120 email questions a month. Volume is tracked by one of the ERI 
librarians who is designated to keep all the question/answer sets that accumulate during the 
month and to count them up before deleting them. Of the total number of questions received, the 
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supervisor estimates 10-15% are repeat users and roughly 1/3 come from out of state. (These are 
general estimates based on his “sense” of the email activity.) 

The official turnaround policy for email reference is 24 hours, but in most cases it is 
actually 3-4 hours. Both print based and electronic resources are used to respond to queries. The 
ERI supervisor says that every question receives a response. There is no calculated percent of 
correct answers; rather quality control is inherent in the peer review process that includes a 
review of all question/answer sets by the supervisor. He states that it is very rare that he needs to 
correct a response, but on occasion he has done so. It should also be noted that standards are in 
place for what an email reference response should look like that includes canned phrases that 
every response is expected to make use of. Librarians also keep on their desktops copies of 
question/answer sets that are considered good examples to guide them. 

As stated above, the question/answer sets are generally discarded instead of being 
archived, but they are used to feed the frequently asked questions on the library’s web site. ERI 
librarians also keep track of useful electronic resources and maintain an internal website for their 
own use, where they share these among themselves. 

In a discussion of what measures would be useful for managing and evaluating the 
service the supervisor showed the most interest in categorizing queries. He feels that 
understanding question type is helpful both for building the reference collection and for training 
staff. For instance, the ERI gets a lot of genealogy questions, which has meant that the ERI staff 
has had to become expert in genealogy resources, even though there are genealogy reference 
librarians in the traditional reference department. Overall however, the volume of questions is so 
low that such assessments are easy to make in an informal way. The other area of evaluation that 
interests this group is customer satisfaction data. They have a plan to review instruments other 
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libraries have created to assess customer satisfaction with email reference and to use these to 
develop their own email survey instrument. 

Marketing 

The QuickAnswer! service is only minimally visible on the library’s web site. A button 
labeled QuickAnswers! is provided on the library’s home page, but the name of the service is not 
fully descriptive of what it is. A user could easily assume that quick answers are a list of 
frequently asked questions or some other librarian prepared finding aid. When this button is 
selected, the user is taken to another page that explains the QuickAnswers! service and provides 
links to the web form, the direct email address, a link for searching the internet, as well as the 
phone number of the call center. So, if the user has not previously book marked the site, he or 
she has to go down three levels to get to the actual input form. Furthermore, this is the only 
pathway in. There are no other links to the web form or indications that this service is offered on 
any of the other library web pages. The QuickAnswer! service is not linked to any other web 
sites they know of. For instance, the service is not available or advertised on Hillsborough 
County web sites or through the school district. 

Publicity for this service has been undertaken in what are fairly traditional library modes. 
For instance, it was written up in the library newsletter and a number of bookmarks (5,000 over 
the 18 months) have been distributed in the library. It is likely that low visibility and minimal 
publicity may be two reasons why the volume of questions received by this service is so low. 
Staffing 

The ERI is essentially a call center staffed by five full time librarians and three part time 
librarians. (One of the part time librarians is an MLS; the other two are library school students. 
All five full time librarians are professional librarians.) The librarians spend most of their time 
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on the telephone answering in the area of 9,000-12,000 calls per month. These librarians are also 
responsible for handling all the email queries that are received. It is easy to see that since email 
queries come in at the rate of three or four a day, they are not a significant drain on the eight 
librarians available to field them. The ERI is open the same hours as the branch library. 

The original staffing for this service, when it started two years ago, resulted from a 
reorganization of personnel and not from new hires. As the (phone) service grew the three new 
librarian positions were created. 

Training for new librarians on how to perform email reference has three stages. The first 
is a two-hour orientation on how to use Pine email and how to format an appropriate email 
response to a query. New librarians then spend one month observing the work of their peers and 
reading through all the question/answer sets generated during that time. After one month they 
begin to pick up a share of the questions and their responses are reviewed by the group and 
corrected as necessary. The most frequent problem is the tendency to over-answer. Reponses to 
email queries tend to take from 10 to 40 minutes to complete, although more involved questions 
have required as much as 1 14 hours. 

Staff performance is measured and evaluated based on the continuous review of their 
question/answer sets. This review includes an assessment of the completeness of the answer, 

' grammar, spelling, and the tone of the message. 

Cost Issues 

This service is provided by the public library funded by Hillsborough County. To date no 
effort has been made calculate the cost of either email or traditional reference services at this 
library. It follows then that measures of cost effectiveness and the performance of a cost/benefit 
analysis have not been undertaken either. It is important to note though that increasingly the 
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county is looking for “business-like results” and that in the next couple of years the library will 
be faced with the need to provide “program based budgets” to the county in order to receive 
funding. 

In anticipation of the new budget process, the Manager of Service and Performance 
Quality has started analyzing the workflow in the library and will be in charge of the work of 
costing out specific programs. This individual was unavailable to meet with the interviewer at 
the time of the site visit, but Gross will contact her by phone as soon as possible to talk to her 
about how she is approaching this problem and what she has learned so far on this topic. 

Future Plans 

There are no current plans to change or significantly enhance the email services provided 
through the ERI department. There are plans underway to redesign the library’s website and the 
issue of the visibility of the QuickAnswer! service is under discussion. There are also plans to 
conduct a user satisfaction survey, but these are only at a very early stage and could not be 
described in much detail. 

The most significant potential change on the horizon is the possible provision of chat 
based reference services as part of the Tampa Bay Library Consortium. These plans are also in 
the beginning stages (at least 18 months away), but there is movement toward purchase of LSSI 
software and the shared provision of chat reference by this group of libraries. 
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Development of Digital Reference and Evaluation Techniques 

The two site visits summarized in this report are part of a larger effort being completed 
by the study team to assess the quality of digital reference service, to produce evaluation tools, 
and determine how best to integrate evaluation efforts into digital reference services. Additional 
site visits have been completed which will eventually be summarized and integrated into the 
report included here. Details about this project can be found at: http://quartz.syr.edu/quality/ . 
Additional information about the study will be provided at the 3 ld VRD Conference to be held 
November 11-13, 2001 in Orlando, http://vrd.org/conferences/VRD2001/ . 

Clearly, the development of digital reference services is in its infancy. Yet generally, 
libraries have not developed ongoing evaluation programs to assess use, users, impacts, and costs 
from such services. As a beginning approach, the study team is considering the development of 
statistics and performance measures based on the following criteria: 

• Extensiveness. How much of a service the is provided (e.g., number of users accessing a 
particular digital reference service, number of digital reference sessions); 

• Efficiency. The use of resources in providing or accessing digital reference services 
(e.g., cost per session in providing access to remote users, average number of times users 
are unable to successfully access the digital reference service); 

• Effectiveness. How well the digital reference service meets the objectives of the provider 
or the user (e.g., success rate of identifying and accessing the information needed by the 
user); 

• Service quality. How well a service or activity is done (e.g., percentage of transactions in 
which users acquire the information they need and are “satisfied” with that information); 



O 

ERiC 



Page 22 



23 



Gross and McClure, August 2001 



Assessing Quality in Digital Reference Services Site Visit Reports 



• Impact. How a digital reference service made a difference in some other activity or 
situation (e.g., the degree to which correct answers enhanced a user’s ability to gain 
employment or pursue business); 

• Usefulness. The degree to which the digital reference services are useful or appropriate 
for individual users (e.g., percentage of services of interest to different types of user 
audiences); and 

• Adoption. The extent to which institutions or users integrate and adopt digital reference 
services into organizational or individual activities (e.g., regularly using digital reference 
services as a normal day-to-day activity). 

These types of criteria provide an important roadmap for thinking about the type of data 
elements and statistics that would be needed to produce such measures, as well as providing a 
quality measurement framework for library networked services and resources. 

As the study progresses specific types of evaluation tools, the study team will develop 
measures, and statistics. Field testing of these measures and statistics may be done at the two 
sites discussed in this report. Ultimately, however, the study’s final report will provide libraries 
with a number of strategies and techniques to conduct on-going formative and summative 
evaluation of digital reference services. 
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